-  . .  Carnegie-Mellon  University 

—  - Software  Engineering  institute 


COB 
0 OBS 
(Mg 

I 

D 

< 


Introduction  to 
Software  Verification  and 
Validation 

Curriculum  Module  SEI-CM-1 3-1.1 


♦ 


DTiC 

El  EC f E 
JUN  0  4  ;39I 


♦ 


* 


/ 


♦ 


♦ 


* 


♦ 


r\ 


♦ 


♦ 


> 


a/ 

91-00847 

IliHiHI 


n^TRIBUTION  STATEMENT  A 

rcved  for  public  ■ 
Distribution  U nKr  : 


91  t>  30 


022 


UNLIMITED,  UNCLASSIFIED 


•  CCU*»*V  CLASSIC  »CA  T  |QN  Of  This  ^*CC 


W  T  S€CU«»T  V  CL-ASSIF  «CAT  ION 

UNCLASSIFIED  _ 


Jf.  SECURITY  CLASSIFICATION  authority  __ 

N/A  _ 


7t>.  OECLASSiF iCATiON'OOWNGRAOiNG  SCmEOULE 

N/A  _ 


4  PERFORMING  ORGANIZATION  REPORT  NUM8EBISI 

SEI-CM-13-I.  I. 


REPORT  DOCUMENTATION  PAGE 


lb.  RESTRICTIVE  MARKINGS 

NONE 


3  O'ST  ri  but  i  on  /  a  v  a  i  la  b  i  li  t  y  of  report 

APPROVED  FOR  PUBLIC  RELEASE 
DISTRIBUTION  UNLIMITED 


S.  MONITORING  ORGANIZATION  ACPORT  NOMBERlS) 


6k  NAME  OP  PERFORMING  ORGANIZATION  Bb.  OFFICE  SYMBOL  I  7a  NAME  OF  MONITORING  ORGANIZATION 

I  Ilf  apphcabla)  | 

SOFTWARE  ENGINEERING  INST.  |  SEI  I  cft  thtnt  pdaceam  r\ rcTrt 


6c.  ADORES S  (City.  $«l(  and  ZIP  Cod*) 

CARNEGIE  MELLON  UNIVERSITY 
PITTSBURGH,  PA  15213 


t*.  NAME  OF  FUNOING/SPONSORiNG 
ORGANIZATION 

SEI  JOINT  PROGRAM  OFFICE 


SEI  JOINT  PROGRAM  OFFICE 


7b.  AOORESS  (City.  Slat a  and  ZIP  Coda)  _ 

ESD/AVS 

HANSCOM  AIR  FORCE  BASE,  MA  01731 


8b.  OFFICE  symbol  a.  procurement  instrument  identification  number 

Ilf  applicant*) 

F1962890C0003 


ESD/  AVS 


to.  SOURCE  OF  FUNOING  NOS. 


CARNEGIE  MELLON  UNIVERSITY 

PROGRAM 

PROJECT 

TASK 

WORK  UNIT 

PITTSBURGH,  PA  15213 

ELEMENT  NO. 

NO. 

NO. 

NO. 

63752F 

N/A 

N/A 

N/A 

1*  TITLE  llncluda  Saeuniy  Clataificaoon) 

itroduction  to  Software  Verification  and  Val 

Ldation 

13.  PERSONAL  AUTMORISt 

James  S.  Collofello, 


13*.  TYPE  OF  REPORT 


I3ticr.ii 


twwifiwiinatiiiPnTif 


13b.  TIME  covereo 
FROM _  TO  . 


14  OATE  OF  REPORT  lYr  .  Mo..  Day) 

December  1988 


is.  PAGE  COUNT 


COS  ATI  COOES 


FiELO  GROUP 


SUB  GR 


It.  SUBJECT  TERMS  iConnnu*  on  ravarta  if  nactuory  and  idandfy  by  block  numbarl 

software  verification  software  technical  review 

software  validation  software  testing 


It.  ABSTRACT  (Conlinua  on  ravaraa  ifntctiMb  ond  idantify  by  block  numbar) 


Software  verification  and  validation  techniques  are  introduced  and  their  applicability 
discussed.  Approaches  to  integrating  these  techinques  into  comprehensive  verification 
and  validation  plans  are  also  addressed.  This  curriculum  module  provides  an  overview 
needed  to  understand  in-depth  curriculum  modules  in  the  verification  and  validation 
area. 


30.  OlSTRI  BUTION/A  VAILABILI T  Y  OF  ABSTRACT 
UNCLASSIPIEO/UNUMITEO  £)  SAME  AS  RPT.  □  .  TIC  USERS  (2 


33a  NAME  op  RESPONSIBLE  INOlVIOUAL 

JOHN  S.  HERMAN.  C*nf.  USAF 


31.  ABSTRACT  SECURITY  CLASSIFICATION 

UNCLASSIFIED,  UNLIMITED  DISTRIBUTION 


33 b.  TELEPHONE. NUMBER 
(Ineluda  Ant*  Coda) 


33c.  OFFICE  SYMBOL 

ESD/AVS 


Introduction  to 

Software  Verification  and  Validation 


SEi  Curriculum  Module  SEI-CM-1 3-1.1 
December  1988 


1  nt+Siioa  f0r 

.oi-io  r.-.-i 

-r-rme-d 

Ju3tij*ic  atlcii. 


James  S.  Coiiofello 
Arizona  State  University 


i  By _ _ 

,'  p_lstruut  ion/ 

| _ Cod 

.Avail  n nd/or 

!  ipeolal 


Di  s  t 


k\ 


! 


Carnegie  Mellon  University 

Software  Engineering  Institute 


This  work  was  sponsored  by  the  U.S.  Department  of  Defense. 
Approved  for  public  release.  Distribution  unlimited. 


This  technical  report  was  prepared  for  the 


SEI  Joint  Program  Office 
ESD/AVS 

Hanscom  AFB,  MA  01731 

The  ideas  and  findings  in  this  report  should  not  be  construed  as  an  official 
DoO  position.  It  is  published  in  the  interest  of  scientific  and  technical 
information  exchange. 

Review  and  Approval 

This  report  has  been  reviewed  and  is  approved  for  publication. 


FOR  THE  COMMANDER 


This  work  is  sponsored  by  the  U.S.  Department  of  Defense. 

Copyright  O  1988  by  Carnegie  Mellon  University. 


This  document  is  avaflable  through  the  Defense  Technical  Information  Center.  DT1C  provides  access  to  and  transfer  of 
scientific  and  technical  information  for  DoD  personnel.  DoD  contractors  and  potential  contractors,  and  other  U.S.  Government 
agency  personnel  and  their  contractors.  To  obtain  a  copy,  please  contact  DT1C  directly:  Defense  Technical  Information 
Center,  Attn:  FDRA,  Cameron  station,  Alexandria.  VA  22304-6145. 

Copies  of  this  document  are  also  avaflable  through  the  National  Technical  Information  Service.  For  information  on  ordering, 
please  contact  NTIS  directly:  National  Technical  Information  Service,  U.S.  Department  of  Commerce,  Springfield,  VA  22161. 

Use  of  any  trademarks  in  this  report  is  not  intended  in  any  way  to  infringe  on  the  rights  of  the  trademark  holder. 


Introduction  to 

Software  Verification  and  Validation 


Foreword 


SEI  curriculum  modules  document  and  explicate  software 
engineering  topics.  They  are  intended  to  be  useful  in  a 
variety  of  situations — in  the  preparation  of  courses,  in  the 
planning  of  individual  lectures,  in  the  design  of  curricula, 
and  in  professional  self-study.  Topics  do  not  exist  in 
isolation,  however,  and  the  question  inevitably  arises  how 
one  module  is  related  to  another.  Because  modules  are 
written  by  different  authors  at  different  times,  the  answer 
could  easily  be  “not  at  all.”  In  a  young  field  struggling  to 
define  itself,  this  would  be  an  unfortunate  situation. 

The  SEI  deliberately  employs  a  number  of  mechanisms  to 
achieve  compatible  paints  of  view  across  curriculum  mod¬ 
ules  and  to  fill  the  content  gaps  between  them.  Modules 
such  as  Introduction  to  Software  Verification  and  Valida¬ 
tion  is  one  of  the  most  important  devices,  hi  this  latest 
revision.  Professor  Collofello  has  more  modules  to  inte¬ 
grate  into  a  coherent  picture  than  when  we  released  his 
first  draft  more  than  a  year  ago — modules  on  quality  as¬ 
surance,  unit  testing,  technical  reviews,  and  formal  verifi¬ 
cation,  as  well  as  less  directly  related  modules  on  specifi¬ 
cation,  requirements  definition,  and  design.  We  believe 
you  will  find  this  curriculum  module  interesting  and  use¬ 
ful,  bath  in  its  own  right  and  by  virtue  of  the  under¬ 
standing  of  other  modules  that  it  facilitates. 

—  Lionel  E.  Deimel 

Senior  Computer  Scientist,  SEI 
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Capsule  Description 


Software  verification  and  validation  techniques  are 
introduced  and  their  applicability  discussed.  Ap¬ 
proaches  to  integrating  these  techniques  into  com¬ 
prehensive  verification  and  validation  plans  are  also 
addressed.  This  curriculum  module  provides  an 
overview  needed  to  understand  in-depth  curriculum 
modules  in  the  verification  and  validation  area. 


Philosophy 


This  module  provides  a  framework  for  understand¬ 
ing  the  application  of  software  verification  and 
validation  (V&V)  processes  throughout  the  software 
evolution  process.  Typical  products  of  this  process 
are  identified,  along  with  their  possible  V&V  objec¬ 
tives.  The  V&V  process  consists  of  numerous  tech¬ 
niques  and  tools,  often  used  in  combination  with  one 
another.  Due  to  the  large  number  of  V&V  ap¬ 
proaches  in  use,  this  module  cannot  address  every 
technique.  Instead,  it  will  analyze  five  categories  of 
V&V  approaches.  These  are: 

•  technical  reviews, 

•  software  testing, 

•  proof  of  correctness  (program  verifica¬ 
tion), 

•  simulation  and  prototyping,  and 

•  requirements  tracing. 

For  each  category,  some  representative  techniques 
will  be  identified  and  assessed.  Since  the  traditional 
focus  of  software  V&V  activity  has  been  software 
testing,  this  category  encompasses  by  far  the  largest 
number  of  techniques.  Thus,  in  this  module,  the 
software  testing  category  will  be  further  refined  to 
expose  the  major  testing  techniques.  Further  depth 
of  coverage  of  techniques  applicable  to  unit  testing 


is  available  in  the  curriculum  module  Unit  Testing 
and  Analysis  [Morell88].  An  additional  module  is 
planned  addressing  integration  and  system  testing  is¬ 
sues.  With  regard  to  the  other  V&V  approaches,  in- 
depth  modules  are  currently  available  on  technical 
review  techniques  ( The  Software  Technical  Review 
Process  [Collofello87])  and  proof  of  correctness 
( Formal  Verification  of  Programs  [Berztiss88]). 

This  module  also  addresses  planning  considerations 
for  V&V  processes,  including  the  selection  and  inte¬ 
gration  of  V&V  techniques  throughout  the  software 
evolution  process. 


Objectives 


This  module  and  its  subordinate,  in-depth  modules, 
is  intended  to  support  the  the  goal  of  preparing  pro¬ 
fessional  software  engineering  students  to  analyze 
the  V&V  objectives  and  concerns  of  a  particular 
project,  examine  the  project’s  constraints,  plan  a 
comprehensive  V&V  strategy  that  includes  the  se¬ 
lection  of  techniques,  track  the  progress  of  the  V&V 
activity,  and  assess  the  effectiveness  of  the  tech¬ 
niques  used  and  that  of  the  overall  V&V  plan.  Typi¬ 
cally,  the  educational  objectives  of  the  software  en¬ 
gineering  educator  teaching  V&V  topics  will  be 
more  modest. 

Possible  objectives  for  the  material  treated  in  this 
curriculum  module  are  given  below. 

The  student  will  be  able  to 

Knowledge 

•  Define  the  terminology  commonly  util¬ 
ized  in  the  verification  and  validation 
area. 

•  Identify  representative  techniques  for  the 
five  categories  of  V&V  approaches. 
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Comprehension 

•  Explain  the  theoretical  and  practical 
limitations  of  V&V  approaches. 

•  Describe  the  V&V  objectives  for  typical 
products  generated  by  the  software 
evolution  process. 

Application 

•  Perform  particular  V&V  techniques. 

Analysis 

•  Determine  the  applicability  and  likely  ef¬ 
fectiveness  of  V&V  approaches  for  par¬ 
ticular  products  of  the  software  evolution 
process. 

Synthesis 

•  Develop  an  outline  for  a  V&V  plan  for  a 
project  that  reflects  understanding  of 
V&V  objectives,  integration  of  tech¬ 
niques,  problem  tracking,  and  assessment 
issues. 

Evaluation 

•  Assess  the  effectiveness  of  a  V&V  plan 
with  respect  to  its  objectives. 


Prerequisite  Knowledge 


This  module  assumes  that  students  have  had  experi¬ 
ence  as  members  of  a  team  working  on  a  substantial 
software  development  project.  This  experience  is 
essential  to  understanding  the  importance  of  soft¬ 
ware  review  techniques  and  integration  approaches 
for  verification  and  validation.  Students  should  also 
understand  the  purpose  of  specifications  and  how  to 
interpret  them.  A  one-semester  software  engineer¬ 
ing  course  can  provide  the  necessary  background. 

Additional  knowledge  that  enhances  the  students’ 
learning  experience  is  that  resulting  from  exposure 
to  a  variety  of  types  of  systems:  real-time  systems, 
expert  systems,  etc.  This  knowledge  may  help  stu¬ 
dents  appreciate  the  complexity  of  V&V  issues,  as 
well  as  provide  them  with  additional  insights  into 
the  applicability  of  various  V&V  approaches. 

Depending  upon  the  level  of  coverage  to  be  ac¬ 
corded  proof  of  correctness,  some  background  in 
formal  logic  may  be  helpful. 


2 


SEI-CM-1 3-1.1 


Introduction  to  Software  Verification  and  Validation 


Module  Content 


Outline _ 

I.  Introduction 

1.  Terminology 

2.  Evolving  Nature  of  Area 
n.  V&V  Limitations 

1.  Theoretical  Foundations 

2.  Impracticality  of  Testing  All  Data 

3.  Impracticality  of  Testing  All  Paths 

4.  No  Absolute  Proof  of  Correctness 

HI.  The  Role  of  V&V  in  Software  Evolution 

1.  Types  of  Products 

a.  Requirements 

b.  Specifications 

c.  Designs 

d.  Implementations 

e.  Changes 

2.  V&V  Objectives 

a.  Correctness 

b.  Consistency 

c.  Necessity 

d.  Sufficiency 

e.  Performance 

IV.  Software  V&V  Approaches  and  their 
Applicability 

1.  Software  Technical  Reviews 

2.  Software  Testing 

a.  Levels  of  Testing 

(i)  Module  Testing 

(ii)  Integration  Testing 

(iii)  System  Testing 

(iv)  Regression  Testing 

b.  Testing  Techniques  and  their  Applicability 

(i)  Functional  Testing  and  Analysis 

(ii)  Structural  Testing  and  Analysis 

(iii)  Error-Oriented  Testing  and  Analysis 

(iv)  Hybrid  Approaches 

(v)  Integration  Strategies 

(vi)  Transaction  Flow  Analysis 

(vii)  Stress  Analysis 


(viii)  Failure  Analysis 
(Lx)  Concurrency  Analysis 
(x)  Performance  Analysis 

3.  Proof  of  Correctness 

4.  Simulation  and  Prototyping 

5.  Requirements  Tracing 
V.  Software  V&V  Planning 

1.  Identification  of  V&V  Goals 

2.  Selection  of  V&V  Techniques 

a.  Requirements 

b.  Specifications 

c.  Designs 

d.  Implementations 

e.  Changes 

3.  Organizational  Responsibilities 

a.  Development  Organization 

b.  Independent  Test  Organization 

c.  Software  Quality  Assurance 

d.  Independent  V&V  Contractor 

4.  Integrating  V&V  Approaches 

5.  Problem  Tracking 

6.  Tracking  Test  Activities 

7.  Assessment 


Annotated  Outline _ 

I.  Introduction 
1.  Terminology 

The  evolution  of  software  that  satisfies  its  user  ex¬ 
pectations  is  a  necessary  goal  of  a  successful  soft¬ 
ware  development  organization.  To  achieve  this 
goal,  software  engineering  practices  must  be  applied 
throughout  the  evolution  of  the  software  product. 
Most  of  these  software  engineering  practices  attempt 
to  create  and  modify  software  in  a  manner  that  max¬ 
imizes  the  probability  of  satisfying  its  user  expec¬ 
tations.  Other  practices,  addressed  in  this  module, 
actually  attempt  to  insure  that  the  product  will  meet 
these  user  expectations.  These  practices  are  collec¬ 
tively  referred  to  as  software  verification  and 
validation  (V&V).  The  reader  is  cautioned  that  ter¬ 
minology  in  this  area  is  often  confusing  and  conflict- 
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ing.  The  glossary  of  this  module  contains  complete 
definitions  of  many  of  the  terms  often  used  to  dis¬ 
cuss  V&V  practices.  This  section  attempts  to  clarify 
terminology  as  it  will  be  used  in  the  remainder  of  the 
module. 

Validation  refers  to  the  process  of  evaluating  soft¬ 
ware  at  the  end  of  its  development  to  insure  that  it  is 
free  from  failures  and  complies  with  its  require¬ 
ments.  A  failure  is  defined  as  incorrect  product  be¬ 
havior.  Often  this  validation  occurs  through  the  util¬ 
ization  of  various  resting  approaches.  Other  inter¬ 
mediate  software  products  may  also  be  validated, 
such  as  the  validation  of  a  requirements  description 
through  the  utilization  of  a  prototype. 

Verification  refers  to  the  process  of  determining 
whether  or  not  the  products  of  a  given  phase  of  a 
software  development  process  fulfill  the  require¬ 
ments  established  during  the  previous  phase.  Soft¬ 
ware  technical  reviews  represent  one  common  ap¬ 
proach  for  verifying  various  products.  For  example, 
a  specifications  review  will  normally  attempt  to  ver¬ 
ify  the  specifications  description  against  a  require¬ 
ments  description  (what  Rombach  has  called  “D- 
requirements”  and  “C-requirements,”  respectively 
[Rombach87]).  Proof  of  correctness  is  another  tech¬ 
nique  for  verifying  programs  to  formal  specifica¬ 
tions.  Verification  approaches  attempt  to  identify 
product  faults  or  errors,  which  give  rise  to  failures. 

2.  Evolving  Nature  of  Area 

As  the  complexity  and  diversity  of  software  prod¬ 
ucts  continue  to  increase,  the  challenge  to  develop 
new  and  more  effective  V&V  strategies  continues. 
The  V&V  approaches  that  were  reasonably  effective 
on  small  batch-oriented  products  are  not  sufficient 
for  concurrent,  distributed,  or  embedded  products. 
Thus,  this  area  will  continue  to  evolve  as  new  re¬ 
search  results  emerge  in  response  to  new  V&V  chal¬ 
lenges. 

n.  V&V  Limitations 

The  overall  objective  of  software  V&V  approaches  is 
to  insure  that  the  product  is  free  from  failures  and 
meets  its  user’s  expectations.  There  are  several  theo¬ 
retical  and  practical  limitations  that  make  this  objective 
impossible  to  obtain  for  many  products. 

1.  Theoretical  Foundations 

Some  of  the  initial  theoretical  foundations  for  testing 
were  presented  by  Goodenough  and  Gerhart  in  then- 
classic  paper  [Goodenough75],  This  paper  provides 
definitions  for  reliability  and  validity,  in  an  attempt 
to  characterize  the  properties  of  a  test  selection  strat¬ 
egy.  A  mathematical  framework  for  investigating 
testing  that  enables  comparisons  of  the  power  of 
testing  methods  is  described  in  [Gourlay83],  How- 
den  claims  the  most  important  theoretical  result  in 
program  testing  and  analysis  is  that  no  general  pur¬ 


pose  testing  or  analysis  procedure  can  be  used  to 
prove  program  correctness.  A  proof  of  this  result  is 
contained  in  his  text  [Howden87], 

2.  Impracticality  of  Testing  All  Data 

For  most  programs,  it  is  impractical  to  auempt  to 
test  the  program  with  all  possible  inputs,  due  to  a 
combinatorial  explosion  [Beizer83,  Howden87],  For 
those  inputs  selected,  a  testing  oracle  is  needed  to 
determine  the  correctness  of  the  output  for  a  partic¬ 
ular  test  input  [Howden87], 

3.  Impracticality  of  Testing  All  Paths 

Far  most  programs,  it  is  impractical  to  attempt  to 
test  all  execution  paths  through  the  product,  due  to  a 
combinatorial  explosion  [B*izer83].  It  is  also  not 
possible  to  develop  an  algorithm  for  generating  test 
data  for  paths  in  an  arbitrary  product,  due  to  the 
inability  to  determine  path  feasibility  [Adrion86], 

4.  No  Absolute  Proof  of  Correctness 

Howden  claims  that  there  is  no  such  thing  as  an 
absolute  proof  of  correctness  [Howden87].  Instead, 
he  suggests  that  there  are  proofs  of  equivalency,  i.e., 
proofs  that  one  description  of  a  product  is  equivalent 
to  another  description.  Hence,  unless  a  formal  spec¬ 
ification  can  be  shown  to  be  correct  and,  indeed, 
reflects  exactly  the  user’s  expectations,  no  claim  of 
product  correctness  can  be  made  [Beizer83,  How- 
dan87], 

ID.  The  Role  of  V&V  in  Software  Evolution 

The  evolution  of  a  software  product  can  proceed  in 
many  ways,  depending  upon  the  development  approach 
used.  The  development  approach  determines  the  spe¬ 
cific  intermediate  products  to  be  created.  For  any 
given  project,  V&V  objectives  must  be  identified  for 
each  of  the  products  created. 

1.  Tvpes  of  Products 

To  simplify  the  discussion  of  V&V  objectives,  five 
types  of  products  are  considered  in  this  module. 
These  types  are  not  meant  to  be  a  partitioning  of  all 
software  documents  and  will  not  be  rigorously  de¬ 
fined.  Within  each  product  type,  many  different 
representational  forms  are  possible.  Each  represen¬ 
tational  form  determines,  to  a  large  extent,  the  ap¬ 
plicability  of  particular  V&V  approaches.  The  in¬ 
tent  here  is  not  to  identify  V&V  approaches  ap¬ 
plicable  to  all  products  in  any  form,  but  instead  to 
describe  V&V  approaches  for  representative  forms 
of  products.  References  are  provided  to  other 
sources  that  treat  particular  approaches  in  depth. 

a.  Requirements 

The  requirements  document  (Rombach  [Rom- 
bach87]:  “customer/user-oriented  requirements” 
or  C-requirements)  provides  an  informal  state¬ 
ment  of  the  user’s  needs. 
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b.  Specifications 

The  specifications  document  (Rombach:  ‘‘design- 
oriented  requirements”  or  D-requirements)  pro¬ 
vides  a  refinement  of  the  user’s  needs,  which 
must  be  satisfied  by  the  product.  There  are  many 
approaches  for  representing  specifications,  both 
formal  and  informal  [Berztiss87,  Rombach87], 

c.  Designs 

The  product  design  describes  how  the  specifica¬ 
tions  will  be  satisfied.  Depending  upon  the  devel¬ 
opment  approach  applied  in  the  project,  there  may 
be  multiple  levels  of  designs.  Numerous  possible 
design  representation  approaches  are  described  in 
Introduction  to  Software  Design  [Budgen88], 

d.  Implementations 

“Implementation”  normally  refers  to  the  source 
code  for  the  product  It  can,  however,  refer  to 
other  implementation-level  products,  such  as  de¬ 
cision  tables  [Beizer83], 

e.  Changes 

Changes  describe  modifications  made  to  die  prod¬ 
uct  Modifications  are  normally  die  result  of  error 
corrections  or  additions  of  new  capabilities  to  the 
product 

2.  V&V  Objectives 

The  specific  V&V  objectives  for  each  product  must 
be  determined  on  a  project-by-project  basis.  This 
determination  will  be  influenced  by  the  criticality  of 
the  product  its  constraints,  and  its  complexity.  In 
general,  the  objective  of  the  V&V  function  is  to  in¬ 
sure  that  the  product  satisfies  the  user  needs.  Thus, 
everything  in  the  product’s  requirements  and  specifi¬ 
cations  must  be  the  target  of  some  V&V  activity.  In 
order  to  limit  the  scope  of  this  module,  however,  the 
V&V  approaches  described  will  concentrate  on  the 
functional  and  performance  portions  of  the  require¬ 
ments  and  specifications  for  the  product.  Ap¬ 
proaches  for  determining  whether  a  product  satisfies 
its  requirements  and  specifications  with  respect  to 
safety,  portability,  usability,  maintainability,  ser¬ 
viceability,  security,  etc.,  although  very  important 
for  many  systems,  will  not  be  addressed  here.  This 
is  consistent  with  the  V&V  approaches  normally  de¬ 
scribed  in  the  literature.  The  broader  picture  of 
“assurance  of  software  quality”  is  addressed  else¬ 
where  [Brown87], 

Limiting  the  scope  of  the  V&V  activities  to  func¬ 
tionality  and  performance,  five  general  V&V  objec¬ 
tives  cm  be  identified  [Howden81 ,  Powell86a], 
These  objectives  provide  a  framework  within  which 
it  is  possible  to  determine  the  applicability  of 
various  V&V  approaches  and  techniques. 


a.  Correctness 

The  extent  to  which  the  product  is  fault  free. 

b.  Consistency 

The  extent  to  which  the  product  is  consistent 
within  itself  and  with  other  products. 

c.  Necessity 

The  extent  to  which  everything  in  die  product  is 
necessary. 

d.  Sufficiency 

The  extent  to  which  the  product  is  complete. 

e.  Performance 

The  extent  to  which  the  product  satisfies  its  per¬ 
formance  requirements. 

TV.  Software  V&V  Approaches  and  their 
Applicability 

Software  V&V  activities  occur  throughout  the  evolu¬ 
tion  of  the  product  There  are  numerous  techniques 
and  tools  that  may  be  used  in  isolation  or  in  combi¬ 
nation  with  each  other.  In  an  effort  to  organize  these 
V&V  activities,  five  broad  classifications  of  ap¬ 
proaches  are  presented.  These  categories  are  not  meant 
to  provide  a  partitioning,  since  there  are  some  tech¬ 
niques  that  span  categories.  Instead,  the  categories  rep¬ 
resent  a  practical  view  that  reflects  the  way  most  of  the 
V&V  approaches  are  described  in  the  literature  and 
used  in  practice.  Possible  combinations  of  these  ap¬ 
proaches  are  discussed  in  the  next  section. 

1.  Software  Technical  Reviews 

The  software  technical  review  process  includes  tech¬ 
niques  such  as  walk-throughs,  inspections,  and 
audits.  Most  of  these  approaches  involve  a  group 
meeting  to  assess  a  work  product  A  comprehensive 
examination  of  the  technical  review  process  and  its 
effectiveness  for  software  products  is  presented  in 
The  Software  Technical  Review  Process  [Collofello- 
88], 

Software  technical  reviews  can  be  used  to  examine 
all  the  products  of  the  software  evolution  process. 
In  particular,  they  are  especially  applicable  and  nec¬ 
essary  for  those  products  not  yet  in  machine- 
processable  form,  such  as  requirements  or  specifi¬ 
cations  written  in  natural  language. 

2.  Software  Testing 

Software  testing  is  the  process  of  exercising  a  prod¬ 
uct  to  verify  that  it  satisfies  specified  requirements 
or  to  identify  differences  between  expected  and  ac¬ 
tual  results  [IEEE83a], 
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a.  Levels  of  Testing 

In  this  section,  various  levels  of  testing  activities, 
each  with  its  own  specific  goals,  are  identified 
and  described.  This  listing  of  levels  is  not  meant 
to  be  complete,  but  will  illustrate  the  notion  of 
levels  of  testing  with  particular  goals.  Other  pos¬ 
sible  levels  of  testing  not  addressed  here  include 
acceptance  testing,  alpha  testing,  beta  testing,  etc. 
[Beizer84], 

(i)  Module  Testing 

Module  (or  unit)  testing  is  the  lowest  level  of 
testing  and  involves  the  testing  of  a  software 
module  or  unit  The  goal  of  module-level  test¬ 
ing  is  to  insure  that  the  component  being  tested 
conforms  to  its  specifications  and  is  ready  to  be 
integrated  with  other  components  of  the  prod¬ 
uct  Module  testing  is  treated  in  depth  in  the 
curriculum  module  Unit  Testing  and  Analysis 
[MorelISS]. 

(ii)  Integration  Testing 

Integration  testing  consists  of  the  systematic 
combination  and  execution  of  product  compo¬ 
nents.  Multiple  levels  of  integration  testing  are 
possible  with  a  combination  of  hardware  and 
software  components  at  several  different 
levels.  The  goal  of  integration  testing  is  to 
insure  that  the  interfaces  between  the  compo¬ 
nents  are  correct  and  that  the  product  compo¬ 
nents  combine  to  execute  the  product’s  func¬ 
tionality  correctly. 

(iii)  System  Testing 

System  testing  is  the  process  of  testing  the  inte¬ 
grated  hardware  and  software  system  to  verify 
that  the  system  meets  its  specified  require¬ 
ments  [IEEE83a].  Practical  priorities  must  be 
established  to  complete  this  task  effectively. 
One  general  priority  is  that  system  testing  must 
concentrate  more  on  ^ /stem  capabilities  rather 
than  component  capabilities  [Beizer84,  Mc- 
Cabe85,  Petschenik85].  This  suggests  that  sys¬ 
tem  tests  concentrate  on  insuring  the  use  and 
interaction  of  functions  rather  than  testing  the 
details  of  their  implementations.  Another  pri¬ 
ority  is  that  testing  typical  situations  is  more 
important  that  testing  special  cases  [Petsche- 
nik85,  Sum86).  This  suggests  that  test  cases  be 
constructed  corresponding  to  high-probability 
user  scenarios.  This  facilitates  early  detection 
of  critical  problems  that  would  greatly  disrupt 
a  user. 

There  are  also  several  key  principles  to  adhere 
to  during  system  testing: 

•  System  tests  should  be  developed  and 
performed  by  a  group  independent  of 
the  people  who  developed  the  code. 


•  System  test  plans  must  be  developed 
and  inspected  with  the  same  rigor  as 
other  elements  of  the  project 

•  System  test  progress  must  be  planned 
and  tracked  similarly  to  other  ele¬ 
ments  of  the  project. 

•  System  tests  must  be  repeatable. 

(iv)  Regression  Testing 

Regression  testing  can  be  defined  as  the  proc¬ 
ess  of  executing  previously  defined  test  cases 
on  a  modified  program  to  assure  that  the  soft¬ 
ware  changes  have  not  adversely  affected  the 
program’s  previously  existing  functions.  The 
error-prone  nature  of  software  modification  de¬ 
mands  that  regression  testing  be  performed. 
Some  examples  of  the  types  of  errors  targeted 
by  regression  testing  include: 

•  Data  corruption  errors.  These  er¬ 
rors  are  side  effects  due  to  shared 
data. 

•  Inappropriate  control  sequencing 
errors.  These  errors  are  side  effects 
due  to  changes  in  execution  se¬ 
quences.  An  example  of  this  type  of 
error  is  the  attempt  to  remove  an  item 
from  a  queue  before  it  is  placed  into 
the  queue. 

•  Resource  contention.  Examples  of 
these  types  of  errors  are  potential 
bottlenecks  and  deadlocks. 

•  Performance  deficiencies.  These 
include  timing  and  storage  utilization 
errors. 

An  important  regression  testing  strategy  is  to 
place  a  higher  priority  on  testing  the  older  ca¬ 
pabilities  of  the  product  than  on  testing  the  new 
capabilities  provided  by  the  modification  [Pet- 
schenik85].  This  insures  that  capabilities  the 
user  has  become  dependent  upon  are  still  in¬ 
tact.  This  is  especially  important  when  we 
consider  that  a  recent  study  found  that  half  of 
all  failures  detected  by  users  after  a  modifi¬ 
cation  were  failures  of  old  capabilities,  as  a 
result  of  side  effects  of  implementation  of  new 
functionality  [Collofello87]. 

Regression  testing  strategies  are  not  well- 
defined  in  the  literature.  They  differ  from  de¬ 
velopment  tests  in  that  development  tests  tend 
to  be  smaller  and  diagnostic  in  nature,  whereas 
regression  tests  tend  to  be  long  and  complex 
scenarios  testing  many  capabilities,  yet  pos¬ 
sibly  proving  unhelpful  in  isolating  a  problem, 
should  one  be  encountered.  Most  regression 
testing  strategies  require  that  some  baseline  of 
product  tests  be  rerun.  These  tests  must  be 


6 


SEI-CM-13-1.1 


Introduction  to  Software  Verification  and  Validation 


supplemented  with  specific  tests  for  the  recent 
modifications.  Strategies  for  testing  modifica¬ 
tions  usually  involve  some  sort  of  systematic 
execution  of  the  modification  and  related  areas. 
At  a  module  level,  this  may  involve  retesting 
module  execution  paths  traversing  the  modifi¬ 
cation.  At  a  product  level,  this  activity  may 
involve  retesting  functions  that  execute  the 
modified  area  [Ftsher77].  The  effectiveness  of 
these  strategies  is  highly  dependent  upon  die 
utilization  of  test  matrices  (see  below),  which 
enable  identification  of  coverage  provided  by 
particular  test  cases. 

b.  Testing  Techniques  and  their  Applicability 
(i)  Functional  Testing  and  Analysis 

Functional  testing  develops  test  data  based 
upon  documents  specifying  the  behavior  of  the 
software.  The  goal  of  functional  testing  is  to 
exercise  each  aspect  of  die  software’s  specified 
behavior  over  some  subset  of  its  input.  How- 
den  has  developed  an  integrated  approach  to 
testing  based  upon  this  notion  of  testing  each 
aspect  of  specified  behavior  [Howd*n86,  How- 
dan87].  A  classification  of  functional  testing 
approaches  and  a  description  of  representative 
techniques  is  presented  in  [Morel  1 88]. 

Functional  testing  and  analysis  techniques  are 
applicable  for  all  levels  of  testing.  However, 
the  level  of  specified  behavior  to  be  tested  will 
normally  be  at  a  higher  level  for  integration 
and  system-level  testing.  Thus,  at  a  module 
level,  it  is  appropriate  to  test  boundary  con¬ 
ditions  and  low-level  functions,  such  as  the 
correct  production  of  a  particular  type  of  error 
message.  At  the  integration  and  system  level, 
the  types  of  functions  tested  are  normally  those 
involving  some  combination  of  lower-level 
functions.  Testing  combinations  of  functions 
involves  selection  of  specific  sequences  of  in¬ 
puts  that  may  reveal  sequencing  errors  due  to: 

•  race  conditions 

•  resource  contention 

•  deadlock 

•  interrupts 

•  synchronization  issues 

Functional  testing  and  analysis  techniques  are 
effective  in  detecting  failures  during  all  levels 
of  testing.  They  must,  however,  be  used  in 
combination  with  other  strategies  to  inprove 
failure  detection  effectiveness  [B«lzer84, 
Girgis86,  HowdenSO,  S«lby86]. 

The  automation  of  functional  testing  tech¬ 
niques  has  been  hampered  by  the  informality  of 
commonly  used  specification  techniques.  The 
difficulty  Iks  in  the  identification  of  the  func¬ 


tions  to  be  tested.  Some  limited  success  in 
automating  this  process  has  been  obtained  for 
some  more  rigorous  specification  techniques. 
These  results  are  described  in  [Morell88]. 

(ii)  Structural  Testing  and  Analysis 

Structural  testing  develops  test  data  based  upon 
the  implementation  of  the  product  Usually 
this  testing  occurs  on  source  code.  However,  it 
is  possible  to  do  structural  testing  on  other 
representations  of  the  program’s  logic.  Struc¬ 
tural  testing  and  analysis  techniques  include 
data  flow  anomaly  detection,  data  flow 
coverage  assessment  and  various  levels  of  path 
coverage.  A  classification  of  structural  testing 
approaches  and  a  description  of  representative 
techniques  is  presented  in  [Morell88]  and  in 
Glenford  Myers’  text  [Myers79], 

Structural  testing  and  analysis  are  applicable  to 
module  testing,  integration  testing,  and  regres¬ 
sion  testing.  At  the  system  test  level,  structural 
testing  is  normally  not  applicable,  due  to  the 
size  of  the  system  to  be  tested.  For  example,  a 
paper  discussing  the  analysis  of  a  product  con¬ 
sisting  of  1.8  million  lines  of  code,  suggests 
that  over  250,000  test  cases  would  be  needed  to 
satisfy  coverage  criteria  [Petschenik85].  At  the 
module  level,  all  of  the  structural  techniques 
are  applicable.  As  the  level  of  testing  increases 
to  the  integration  level,  the  focus  of  the  struc¬ 
tural  techniques  is  on  the  area  of  interface  anal¬ 
ysis  [Howden87],  This  interface  analysis  may 
involve  module  interfaces,  as  well  as  interfaces 
to  other  system  components.  Structural  testing 
and  analysis  can  also  be  performed  on  designs 
using  manual  walk-throughs  or  design  simula¬ 
tions  [Powsll86a]. 

Structural  testing  and  analysis  techniques  are 
very  effective  in  detecting  failures  during  the 
module  and  integration  testing  levels.  Beizer 
reports  that  path  testing  catches  50%  of  all  er¬ 
rors  during  module  testing  and  a  total  of  one- 
third  of  all  of  the  errors  [Beizer84].  Structural 
testing  is  very  cumbersome  to  perform  without 
tools,  and  even  with  tools  requires  considerable 
effort  to  achieve  desirable  levels  of  coverage. 
Since  structural  testing  and  analysis  techniques 
cannot  detect  missing  functions  (nor  some 
other  types  of  errors),  they  must  be  used  in 
combination  with  other  strategies  to  improve 
failure  detection  effectiveness  [Beizer84, 
Girgis86,  HowdenQO,  Selby86], 

There  are  numerous  automated  techniques  to 
support  structural  testing  and  analysis.  Most  of 
the  automated  approaches  provide  statement 
and  branch  coverage.  Tools  for  automating 
several  structural  testing  techniques  are  de¬ 
scribed  in  the  papers  cited  in  [Morell88], 
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(iii)  Error-Oriented  Testing  and  Analysis 

Error-oriented  testing  and  analysis  techniques 
are  those  that  focus  on  the  presence  or  absense 
of  errors  in  the  programming  process.  A  clas¬ 
sification  of  these  approaches  and  a  description 
of  representative  techniques  is  presented  in 
[Mor*ll88]. 

Error-oriented  testing  and  analysis  techniques 
are,  in  general,  applicable  to  all  levels  of  test¬ 
ing.  Some  techniques,  such  as  statistical  meth¬ 
ods  [Curritae],  error  seeding  [M!lts83],  and 
mutation  testing  [DeMillo78],  are  particularly 
suited  to  application  during  the  integration  and 
system  levels  of  testing. 

(iv)  Hybrid  Approaches 

Combinations  of  the  functional,  structural,  and 
error-oriented  techniques  have  been  investi¬ 
gated  and  are  described  in  [Morall86].  These 
hybrid  approaches  involve  integration  of  tech¬ 
niques,  rather  than  their  composition.  Hybrid 
approaches,  particularly  those  involving  struc¬ 
tural  testing,  are  normally  applicable  at  the 
module  level. 

(v)  Integration  Strategies 

Integration  consists  of  the  systematic  combi¬ 
nation  and  analysis  of  product  components.  It 
is  assumed  that  the  components  being  inte¬ 
grated  have  already  been  individually  ex¬ 
amined  for  correctness.  This  insures  that  the 
emphasis  of  the  integration  activity  is  on  ex¬ 
amining  the  interaction  of  the  components 
[BaizarM,  Howdan87].  Although  integration 
strategies  are  normally  discussed  for  imple¬ 
mentations,  they  are  also  applicable  for  inte¬ 
grating  the  components  of  any  product,  such  as 
designs. 

There  are  several  types  of  errors  targeted  by 
integration  testing: 

•  Import/export  range  errors  This 
type  of  error  occurs  when  the  source 
of  input  parameters  falls  outside  of 
the  range  of  their  destination.  For 
example,  assume  module  A  calls 
module  B  with  table  pointer  X.  If  A 
assumes  a  maximum  table  size  of  10 
and  B  assumes  a  maximum  table  size 
of  8,  an  import/export  range  error  oc¬ 
curs.  The  detection  of  this  type  of 
error  requires  careful  boundary-value 
testing  of  parameters. 

•  Import/export  type  compatibility 
errors.  This  type  of  error  is  attri¬ 
buted  to  a  mismatch  of  user-defined 
types.  These  errors  are  normally  de¬ 
tected  by  compilers  or  code  inspec¬ 
tions. 


•  Import/export  representation  er¬ 
rors.  This  type  of  error  occurs  when 
parameters  are  of  the  same  type,  but 
the  meaning  of  the  parameters  is  dif¬ 
ferent  in  the  calling  and  called  mod¬ 
ules.  For  example,  assume  module  A 
passes  a  parameter  ElapsedJTime,  of 
type  real,  to  module  B.  Module  A 
might  pass  the  value  as  seconds, 
while  module  B  is  assuming  the 
value  is  passed  as  milliseconds. 

These  types  of  errors  are  difficult  to 
detect,  although  range  checks  and  in¬ 
spections  provide  some  assistance. 

•  Parameter  utilization  errors.  Dan¬ 
gerous  assumptions  are  often  made 
concerning  whether  a  module  called 
will  alter  the  information  passed  to  it 
Although  support  for  detecting  such 
errors  is  provided  by  some  compilers, 
careful  testing  and/or  inspections 
may  be  necessary  to  insure  that 
values  have  not  been  unexpectedly 
corrupted. 

•  Integration  time  domain/  computa¬ 
tion  errors.  A  domain  error  occurs 
when  a  specific  input  follows  the 
wrong  path  due  to  an  error  in  the 
control  flow.  A  computation  error 
exists  when  a  specific  input  follows 
the  correct  path,  but  an  error  in  some 
assignment  statement  causes  the 
wrong  function  to  be  computed.  Al¬ 
though  domain  and  computation  er¬ 
rors  are  normally  addressed  during 
module  testing,  the  concepts  apply 
across  module  boundaries.  In  fact, 
some  domain  and  computation  errors 
in  the  integrated  program  might  be 
masked  during  integration  testing  if 
the  module  being  integrated  is  as¬ 
sumed  to  be  correct  and  is  treated  as 
a  blade  box.  Examples  of  these  types 
of  errors  and  an  approach  for  detect¬ 
ing  them  is  presented  in  [Haley84], 

Measures  of  integration  coverage  can  be  de¬ 
fined  in  an  analogous  way  to  those  defined  by 
Miller  [Mill*r77]  for  module-level  coverage. 
Whereas  Miller’s  “Cl”  measure  requires  every 
statement  to  be  executed,  an  *11”  measure  for 
integration  coverage  might  require  every  mod¬ 
ule  to  be  invoked  during  the  integration  test 
The  **C2”  measure,  which  requires  each  branch 
to  be  executed,  might  have  an  integration 
coverage  counterpart  *T2”  that  requires  each 
module  to  be  invoked  by  all  possible  callers. 
An  “13”  measure  might  require  that  each  call  in 
every  module  be  executed. 
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In  addition  to  module-level  coverage,  various 
interface  coverage  measures  can  be  defined. 
An  “XO”  measure  requires  each  I/O  interface 
be  utilized.  This  implies  that  passed  parame¬ 
ters  are  referenced,  as  well  as  returned,  param¬ 
eters.  An  “XI”  measure  requires  that  each  out¬ 
put  variable  of  a  module  be  set  in  all  possible 
assignments  and  that  each  input  variable  be 
used  at  all  possible  reference  points. 

Several  strategies  for  integration  testing  exist 
These  strategies  may  be  used  independently  or 
in  combination.  The  primary  techniques  are 
top-down,  bottom-up,  big-bang,  and  threaded 
integration,  although  terminology  used  in  the 
literature  varie  Top-down  integration  at¬ 
tempts  to  com^i r  e  incrementally  the  compo¬ 
nents  of  the  program,  starting  with  the  topmost 
element  and  simulating  lower  level  elements 
with  stubs.  Each  stub  is  then  replaced  with  an 
actual  program  component  as  the  integration 
process  proceeds  in  a  top-down  fashion.  Top- 
down  integration  is  useful  for  those  compo¬ 
nents  of  the  program  with  complicated  control 
structures  [B«izer84].  It  also  provides  visibility 
into  the  integration  process  by  demonstrating  a 
potentially  useful  product  early. 

Bottom-up  integration  attempts  to  combine  in¬ 
crementally  components  of  the  program  start¬ 
ing  with  those  components  that  do  not  invoke 
other  components.  Test  drivers  must  be  con¬ 
structed  to  invoke  these  components.  As 
bottom-up  integration  proceeds,  test  drivers  are 
replaced  with  the  actual  program  components 
that  perform  the  invocation,  and  new  test 
drivers  are  constructed  until  the  "top”  of  the 
program  is  readied.  Bottom-up  integration  is 
consistent  with  the  notion  of  developing  soft¬ 
ware  as  a  series  of  building  blocks.  Bottom-up 
integration  should  proceed  wherever  the  driv¬ 
ing  control  structure  is  not  too  complicated 
[Beizer84]. 

Big-bang  integration  is  not  an  incremental 
strategy  and  involves  combining  and  testing  all 
modules  at  once.  Except  for  small  programs, 
big-bang  integration  is  not  a  cost-effective 
technique  because  of  the  difficulty  of  isolating 
integration  testing  failures  [Beizer84], 

Threaded  integration  is  an  incremental  tech¬ 
nique  that  identifies  major  processing  functions 
that  the  product  is  to  perform  and  maps  these 
functions  to  modules  implementing  them. 
Each  processing  function  is  called  a  thread.  A 
collection  of  related  threads  is  often  called  a 
build.  Builds  may  serve  as  a  basis  for  test 
management  To  test  a  thread,  the  group  of 
modules  corresponding  to  the  thread  is  com¬ 
bined.  For  those  modules  in  the  thread  with 


interfaces  to  other  modules  not  supporting  the 
thread,  stubs  are  used.  The  initial  threads  to  be 
tested  normally  correspond  to  the  “backbone” 
or  “skeleton”  of  the  product  under  test.  (These 
terms  are  also  used  to  refer  to  this  type  of  inte¬ 
gration  strategy.)  The  addition  of  new  threads 
for  the  product  undergoing  integration  pro¬ 
ceeds  incrementally  in  a  planned  fashion.  The 
use  of  system  verification  diagrams  for 
“threading”  requirements  is  described  in 
[Deutsch82]. 

(vi)  Transaction  Flow  Analysis 

Transaction  flow  analysis  develops  test  data  to 
execute  sequences  of  tasks  that  correspond  to  a 
transaction,  where  a  “transaction”  is  defined  as 
a  unit  of  work  seen  from  a  system  user’s  point 
of  view  [Beizer84,  McCabe85,  Petschenik85]. 
An  example  of  a  transaction  for  an  operating 
system  might  be  a  request  to  print  a  file.  The 
execution  of  this  transaction  requires  several 
tasks,  such  as  checking  the  existence  of  the 
file,  validating  permission  to  read  the  file,  etc. 

The  first  step  of  transaction  flow  analysis  is  to 
identify  the  transactions.  McCabe  suggests  the 
drawing  of  data  flow  diagrams  after  integration 
testing  to  model  the  logical  flow  of  the  system. 
Each  transaction  can  then  be  identified  as  a 
path  through  the  data  flow  diagram,  with  each 
data  flow  process  corresponding  to  a  task  that 
must  be  tested  in  combination  with  other  tasks 
on  the  transaction  flow  [McCabe85],  Informa¬ 
tion  about  transaction  flows  may  also  be  ob¬ 
tained  from  HIPO  diagrams,  Petri  nets,  or  other 
similar  system-level  documentation  [B«izer84]. 

Once  die  transaction  flows  have  been  identi¬ 
fied,  black-box  testing  techniques  can  be  util¬ 
ized  to  generate  test  data  for  selected  paths 
through  the  transaction  flow  diagram.  Some 
possible  guidelines  for  selecting  paths  follow: 

•  Test  every  link/decision  in  the  trans¬ 
action  flow  graph. 

•  Test  each  loop  with  a  single,  double, 
typical,  maximum,  and  maximum¬ 
less-one  number  of  iterations. 

•  Test  combinations  of  paths  within 
and  between  transaction  flows. 

•  Test  that  the  system  does  not  do 
things  that  it  is  not  supposed  to  do, 
by  watching  for  unexpected  se¬ 
quences  of  paths  within  and  between 
transaction  flows. 

Transaction  flow  analysis  is  a  very  effective 
technique  for  identifying  errors  corresponding 
to  interface  problems  with  functional  tasks.  It 
is  most  applicable  to  integration  and  system- 
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level  testing.  The  technique  is  also  appropriate 
for  addressing  completeness  and  correctness  is¬ 
sues  for  requirements,  specifications,  and  de¬ 
signs. 

(vii)  Stress  Analysis 

Stress  analysis  involves  analyzing  die  behavior 
of  the  system  when  its  resources  are  saturated, 
in  order  to  assess  whether  or  not  the  system 
will  continue  to  satisfy  its  specifications. 
Some  examples  of  errors  targeted  by  stress 
tests  include: 

•  potential  race  conditions 

•  errors  in  processing  sequences 

•  errors  in  limits,  thresholds,  or  con¬ 
trols  designed  to  deal  with  overload 
situations 

•  resource  contention  and  depletion 

For  example,  one  typical  stress  test  for  an 
operating  system  would  be  a  program  that  re¬ 
quests  as  much  memory  as  the  system  has 
available. 

The  first  step  in  performing  a  stress  analysis  is 
identifying  those  resources  that  can  and  should 
be  stressed.  This  identification  is  very  system  - 
dependent,  but  often  includes  resources  such  as 
file  space,  memory,  I/O  buffers,  processing 
time,  and  interrupt  handlers.  Once  these 
resources  have  been  identified,  test  cases  must 
be  designed  to  stress  them.  These  tests  often 
require  large  amounts  of  data,  for  which  auto¬ 
mated  support  in  the  form  of  test-case  genera¬ 
tors  is  needed  [Baizar64,  Sum86], 

Although  stress  analysis  is  often  viewed  as  one 
of  the  last  tasks  to  be  performed  during  system 
testing,  it  is  most  effective  if  it  is  applied  dur¬ 
ing  each  of  the  product’s  V&V  activities. 
Many  of  the  errors  detected  during  a  stress 
analysis  correspond  to  serious  design  flaws. 
For  example,  a  stress  analysis  of  a  design  may 
involve  an  identification  of  potential  bot¬ 
tlenecks  that  may  prevent  the  product  from  sat¬ 
isfying  its  specifications  under  extreme  loads 
[Beizer84], 

Stress  analysis  is  a  necessary  complement  to 
the  previously  described  testing  and  analysis 
techniques  for  resource-critical  applications. 
Whereas  the  foregoing  techniques  primarily 
view  the  product  under  normal  operating  con¬ 
ditions,  stress  analysis  views  the  product  under 
conditions  that  may  not  have  been  anticipated. 
Stress  analysis  techniques  can  also  be  com¬ 
bined  with  other  approaches  during  V&V  acti¬ 
vities  to  insure  that  the  product’s  specifications 
for  such  attributes  as  performance,  safety, 
security,  etc.,  are  met 


(viii)  Failure  Analysis 

Failure  analysis  is  die  examination  of  the 
product’s  reaction  to  failures  of  hardware  or 
software.  The  product’s  specifications  must  be 
examined  to  determine  precisely  which  types 
of  failures  must  be  analyzed  and  what  the 
product’s  reaction  must  be.  Failure  analysis  is 
sometimes  referred  to  as  “recovery  testing” 
[Beizer84]. 

Failure  analysis  must  be  performed  during  each 
of  the  product’s  V&V  activities.  It  is  essential 
during  requirement  and  specification  V&V  ac¬ 
tivities  that  a  clear  statement  of  die  product’s 
response  to  various  types  of  failures  be  ad¬ 
dressed  in  terms  that  allow  analysis.  The  de¬ 
sign  must  also  be  analyzed  to  show  that  the 
product’s  reaction  to  failures  satisfies  its  speci¬ 
fications.  The  failure  analysis  of  implemen¬ 
tations  often  occurs  during  system  testing. 
This  testing  may  take  the  form  of  simulating 
hardware  or  software  errors  or  actual  introduc¬ 
tion  of  these  types  of  errors. 

Failure  analysis  is  essential  to  detecting  prod¬ 
uct  recovery  errors.  These  errors  can  lead  to 
lost  files,  lost  data,  duplicate  transactions,  etc. 
Failure  analysis  techniques  can  also  be  com¬ 
bined  with  other  approaches  during  V&V  acti¬ 
vities  to  insure  that  the  product’s  specifications 
for  such  attributes  as  performance,  security, 
safety,  usability,  etc.,  are  met 

(ix)  Concurrency  Analysis 

Concurrency  analysis  examines  the  interaction 
of  tasks  being  executed  simultaneously  within 
the  product  to  insure  that  the  overall  specifi¬ 
cations  are  being  met  Concurrent  tasks  may 
be  executed  in  parallel  or  have  their  execution 
interleaved.  Concurrency  analysis  is  some¬ 
times  referred  to  as  “background  testing” 
[Beizer84], 

For  products  with  tasks  that  may  execute  in 
parallel,  concurrency  analysis  must  be  per¬ 
formed  during  each  of  the  product’s  V&V  acti¬ 
vities.  During  design,  concurrency  analysis 
should  be  performed  to  identify  such  issues  as 
potential  contention  for  resources,  deadlock, 
and  priorities.  A  concurrency  analysis  for  im¬ 
plementations  normally  takes  place  during  sys¬ 
tem  testing.  Tests  must  be  designed,  executed, 
and  analyzed  to  exploit  the  parallelism  in  the 
system  and  insure  that  the  specifications  are 
met 

(x)  Performance  Analysis 

The  goal  of  performance  analysis  is  to  insure 
that  the  product  meets  its  specified  perfor- 
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mance  objectives.  These  objectives  must  be 
stated  in  measurable  terms,  so  far  as  possible. 
Typical  performance  objectives  relate  to  re¬ 
sponse  time  and  system  throughput  [Beizer84], 

A  performance  analysis  should  be  applied  dur¬ 
ing  each  of  the  product’s  V&V  activities.  Dur¬ 
ing  requirement  and  specification  V&V  activi¬ 
ties,  performance  objectives  must  be  analyzed 
to  insure  completeness,  feasibility,  and  tes¬ 
tability.  Prototyping,  simulation,  or  other 
modeling  approaches  may  be  used  to  insure 
feasibility.  For  designs,  the  performance  re¬ 
quirements  must  be  allocated  to  individual 
components.  These  components  can  then  be 
analyzed  to  determine  if  the  performance  re¬ 
quirements  can  be  met  Prototyping,  simula¬ 
tion,  and  other  modeling  approaches  again  are 
techniques  applicable  to  this  task.  For  imple¬ 
mentations,  a  performance  analysis  can  take 
place  during  each  level  of  testing.  Test  data 
must  be  carefully  constructed  to  correspond  to 
the  scenarios  for  which  the  performance  re¬ 
quirements  were  specified. 

3.  Proof  of  Correctness 

Proof  of  correctness  is  a  collection  of  techniques  that 
apply  the  formality  and  rigor  of  mathematics  to  the 
task  of  proving  the  consistency  between  an  algorith¬ 
mic  solution  and  a  rigorous,  complete  specification 
of  the  intent  of  the  solution  [Adrion86,  Powell86b], 
This  technique  is  also  often  referred  to  as  “formal 
verification.”  The  usual  proof  technique  follows 
Floyd's  Method  of  Inductive  Assertions  or  some 
variant  [Floyd67,  Hantler76]. 

Proof  of  correctness  techniques  are  normally 
presented  in  the  context  of  verifying  an  implemen¬ 
tation  against  a  specification.  The  techniques  are 
also  applicable  in  verifying  the  correctness  of  other 
products,  as  long  as  they  possess  a  formal  represen¬ 
tation  [Ambler78,  Korelsky87], 

There  are  several  limitations  to  proof  of  correctness 
techniques.  One  limitation  is  the  dependence  of  the 
technique  upon  a  correct  formal  specification  that 
reflects  the  user’s  needs.  Current  specification  ap¬ 
proaches  cannot  always  capture  these  needs  in  a  for¬ 
mal  way,  especially  when  product  aspects  such  as 
performance,  reliability,  quality,  etc.,  are  considered 
[B«rztiss87,  Rombach87],  Another  limitation  has  to 
do  with  the  complexity  of  rigorously  specifying  the 
execution  behavior  of  the  computing  environment. 
For  large  programs,  the  amount  of  detail  to  handle, 
combined  with  the  lack  of  powerful  tools  may  make 
the  proof  technique  impractical  [Beizer83,  Korelsky- 
87,  Howden87,  Pow®  1186b], 

More  information  on  proof  of  correctness  ap¬ 
proaches  is  contained  in  the  curriculum  module 
Formal  Verification  of  Programs  [Berztisa88]. 


4.  Simulation  and  Prototyping 

Simulation  and  prototyping  are  techniques  for 
analyzing  the  expected  behavior  of  a  product  There 
are  many  approaches  to  constructing  simulations  and 
prototypes  that  are  well-documented  in  the  litera¬ 
ture.  For  V&V  purposes,  simulations  and  proto¬ 
types  are  normally  used  to  analyze  requirements  and 
specifications  to  insure  that  they  reflect  the  user’s 
needs  [Brackett88].  Since  they  are  executable,  they 
offer  additional  insight  into  the  completeness  and 
correctness  of  these  documents.  Simulations  and 
prototypes  can  also  be  used  to  analyze  predicted 
product  performance,  especially  for  candidate  prod¬ 
uct  designs,  to  insure  that  they  conform  to  the  re¬ 
quirements.  It  is  important  to  note  that  the  utili¬ 
zation  of  simulation  and  prototyping  as  V&V  tech¬ 
niques  requires  that  the  simulations  and  prototypes 
themselves  be  correct.  Thus,  the  utilization  of  these 
techniques  requires  an  additional  level  of  V&V  acti¬ 
vity. 

5.  Requirements  Tracing 

Requirements  tracing  is  a  technique  for  insuring  that 
the  product,  as  well  as  the  testing  of  the  product, 
addresses  each  of  its  requirements.  The  usual  ap¬ 
proach  to  performing  requirements  tracing  uses 
matrices.  One  type  of  matrix  maps  requirements  to 
software  modules.  Construction  and  analysis  of  this 
matrix  can  help  insure  that  all  requirements  are 
properly  addressed  by  the  product  and  that  the  prod¬ 
uct  does  not  have  any  superfluous  capabilities 
[Powell86b].  System  Verification  Diagrams  are 
another  way  of  analyzing  requirements  An  odules 
traceability  [Deutsch82].  Another  type  of  matrix 
maps  requirements  to  test  cases.  Construction  and 
analysis  of  this  matrix  can  help  insure  that  all  re¬ 
quirements  are  properly  tested.  A  third  type  of 
matrix  maps  requirements  to  their  evaluation  ap¬ 
proach.  The  evaluation  approaches  may  consist  of 
various  levels  of  testing,  reviews,  simulations,  etc. 
The  requirements/evaluation  matrix  insures  that  all 
requirements  will  undergo  some  form  of  V&V 
[Deutsch82,  Powell86b].  Requirements  tracing  can 
be  applied  for  all  of  the  products  of  the  software 
evolution  process. 

V.  Software  V&V  Planning 

The  development  of  a  comprehensive  V&V  plan  is  es¬ 
sential  to  the  success  of  a  project  This  plan  must  be 
developed  early  in  the  project  Depending  on  the  de¬ 
velopment  approach  followed,  multiple  levels  of  test 
plans  may  be  developed,  corresponding  to  various 
levels  of  V&V  activities.  Guidelines  for  the  contents 
of  system,  software,  build,  and  module  test  plans  have 
been  documented  in  the  literature  [Deutsch82,  DoD87, 
Evans84,  NBS76,  IEEE83b].  These  references  also 
contain  suggestions  about  bow  to  document  other  in¬ 
formation,  such  as  test  procedures  and  test  cases.  The 
formulation  of  an  effective  V&V  plan  requires  many 
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considerations  that  are  defined  in  the  remainder  of  this 
section. 

1.  Identification  of  V&V  Goals 

V&V  goals  must  be  identified  from  the  requirements 
and  specifications.  These  goals  must  address  those 
attributes  of  the  product  that  correspond  to  its  user 
expectations.  These  goals  must  be  achievable, 
taking  into  account  both  theoretical  and  practical 
limitations  [Evans84,  Powell86a,  Sum86], 

2.  Selection  of  V&V  Techniques 

Once  a  set  of  V&V  objectives  has  been  identified, 
specific  techniques  must  be  selected  for  each  of  the 
project’s  evolving  products.  A  methodology  for  die 
selection  of  techniques  and  tools  is  presented  in 
[Powell86b].  More  specific  guidelines  for  the  selec¬ 
tion  of  techniques  applicable  at  the  unit  level  of  test¬ 
ing  are  presented  in  [Morell08].  A  mapping  of  some 
of  the  approaches  presented  in  Section  IV  of  this 
module  to  the  products  in  Section  III  follows. 

a.  Requirements 

The  applicable  techniques  for  accomplishing  the 
V&V  objectives  for  requirements  are  technical  re¬ 
views,  prototyping,  and  simulation.  The  review 
process  is  often  called  a  System  Requirements 
Review  (SRR).  Depending  upon  the  represen¬ 
tation  of  the  requirements,  consistency  analyzers 
may  be  used  to  support  the  SRR. 

b.  Specifications 

The  applicable  techniques  for  accomplishing  die 
V&V  objectives  far  specifications  are  technical 
reviews,  requirements  tracing,  prototyping,  and 
simulation.  The  specifications  review  is  some¬ 
times  combined  with  a  review  of  the  product’s 
high-level  design.  The  requirements  must  be 
traced  to  the  specifications. 

c.  Designs 

The  applicable  techniques  for  accomplishing  the 
V&V  objectives  for  designs  are  technical  reviews, 
requirements  tracing,  prototyping,  simulation,  and 
proof  of  correctness.  High-level  designs  that  cor¬ 
respond  to  an  architectural  view  of  the  product 
are  often  reviewed  in  a  Preliminary  Design  Re¬ 
view.  Detailed  designs  are  addressed  by  a  Criti¬ 
cal  Design  Review.  Depending  upon  the  repre¬ 
sentation  of  the  design,  static  analyzers  may  be 
used  to  assist  these  review  processes.  Require¬ 
ments  must  be  traced  to  modules  in  the  architec¬ 
tural  design;  matrices  can  be  used  to  facilitate  this 
process  [Pcw«ll86b].  Prototyping  and  simulation 
can  be  used  to  assess  feasibility  and  adherence  to 
performance  requirements.  Proofs  of  correctness, 
where  applicable,  are  normally  performed  at  the 
detailed  design  level  [Dysr87]. 


d  Implementations 

The  applicable  techniques  for  accomplishing  the 
V&V  objectives  for  implementations  are  techni¬ 
cal  reviews,  requirements  tracing,  testing,  and 
proof  of  correctness.  Various  code  review  tech¬ 
niques  such  as  walk-throughs  and  inspections  ex¬ 
ist  At  the  source-code  level,  several  static  anal¬ 
ysis  techniques  are  available  for  detecting  imple¬ 
mentation  errors.  The  requirements  tracing  acti¬ 
vity  is  here  concerned  with  tracing  requirements 
to  source-code  modules.  The  bulk  of  die  V&V 
activity  for  source  code  consists  of  testing.  Multi¬ 
ple  levels  of  testing  are  usually  performed 
Where  applicable,  proof-of-correctness  tech¬ 
niques  may  be  applied  usually  at  the  module 
level. 

e.  Changes 

Since  changes  describe  modifications  to  products, 
the  same  techniques  used  for  V&V  during  devel¬ 
opment  may  be  applied  during  modification. 
Changes  to  implementations  require  regression 
testing. 

3.  Organizational  Responsibilities 

The  organizational  structure  of  a  project  is  a  key 
planning  consideration  for  project  managers.  An 
important  aspect  of  this  structure  is  delegation  of 
V&V  activities  to  various  organizations  [Deutsch82, 
Evans84,  Petschenik85,  Sum86],  This  decision  is 
often  based  upon  the  size,  complexity,  and  criticality 
of  the  product  In  this  module,  four  types  of  organi¬ 
zations  are  addressed  These  organizations  reflect 
typical  strategies  for  partitioning  tasks  to  achieve 
V&V  goals  for  the  product.  It  is,  of  course,  possible 
to  delegate  these  V&V  activities  in  many  other 
ways. 

a.  Development  Organization 

The  development  organization  has  responsibility 
for  participating  in  technical  reviews  for  all  of  tire 
evolution  products.  These  reviews  must  insure 
that  the  requirements  can  be  traced  throughout  the 
class  of  products.  The  development  organization 
may  also  construct  prototypes  and  simulations. 
For  code,  the  development  organization  has  re¬ 
sponsibility  for  preparing  and  executing  test  plans 
for  unit  and  integration  levels  of  testing.  In  some 
environments,  this  is  referred  to  as  Preliminary 
Qualification  Testing.  The  development  organi¬ 
zation  also  constructs  any  applicable  proofs  of 
correctness  at  the  module  level. 

b.  Independent  Test  Organization 

An  independent  test  organization  (ITO)  may  be 
established  due  to  the  magnitude  of  the  testing 
effort  or  the  need  for  objectivity.  An  ITO  enables 
the  preparation  for  test  activities  to  occur  in  paral- 
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lei  with  those  of  development.  The  ITO  normally 
participates  in  all  of  the  product’s  technical  re¬ 
views  and  monitors  the  preliminary  qualification 
testing  effort.  The  primary  responsibility  of  the 
ITO  is  the  preparation  and  execution  of  the 
product’s  system  test  plan.  This  is  sometimes 
referred  to  as  the  Formal  Qualification  Test  The 
plan  for  this  must  contain  the  equivalent  of  a 
requirements/evaluation  matrix  that  defines  the 
V&V  approach  to  be  applied  for  each  requirement 
[Deutsch82].  If  the  product  must  be  integrated 
with  other  products,  this  integration  activity  is 
normally  the  responsibility  of  the  ITO  as  well 

c.  Software  Quality  Assurance 

Although  software  quality  assurance  may  exist  as 
a  separate  organization,  die  intent  here  is  to  iden¬ 
tify  some  activities  for  assuring  software  quality 
that  may  be  distributed  using  any  of  a  number  of 
organizational  structures  [Brown87],  Evaluations 
are  the  primary  avenue  for  assuring  software 
quality.  Some  typical  types  of  evaluations  to  be 
performed  where  appropriate  throughout  die 
product  life  cycle  are  identified  below.  Other 
types  can  be  found  in  Assurance  of  Software 
Quality  [Brown87],  Evaluation  types: 

•  internal  consistency  of  product 

•  understandability  of  product 

•  traceability  to  indicated  documents 

•  consistency  with  indicated  documents 

•  appropriate  allocation  of  sizing,  timing 
resources 

•  adequate  test  coverage  of  requirements 

•  consistency  between  data  definitions 
and  use 

•  adequacy  of  test  cases  and  test  proce¬ 
dures 

•  completeness  of  testing 

•  completeness  of  regression  testing 

d.  Independent  V&V  Contractor 

An  independent  V&V  contractor  may  sometimes 
be  used  to  insure  independent  objectivity  and 
evaluation  for  the  customer.  The  scope  of  activi¬ 
ties  for  this  contractor  varies,  including  any  or  all 
of  the  activities  addressed  for  the  Independent 
Test  and  Software  Quality  Assurance  organiza¬ 
tions  [D«utach82]. 

4.  Integrating  V&V  Approaches 

Once  a  set  of  V&V  objectives  has  been  identified, 
an  overall  integrated  V&V  approach  must  be  deter¬ 
mined.  This  approach  involves  integration  of  tech¬ 
niques  applicable  to  the  various  life  cycle  phases  as 
well  as  delegation  of  these  tasks  among  the  project’s 
organizations.  The  planning  of  this  integrated  V&V 


approach  is  very  dependent  upon  the  nature  of  the 
product  and  die  process  used  to  develop  it  Tradi¬ 
tional  integrated  V&V  approaches  have  followed  the 
“waterfall  model”  with  various  V&V  functions  al¬ 
located  to  the  project’s  development  phases 
[Deutsch82,  DoD87,  Evans84,  Powell86a].  Alter¬ 
natives  to  this  approach  exist  such  as  the  Qeanroom 
software  development  process  developed  by  IBM. 
This  approach  is  based  on  a  software  development 
process  that  produces  incremental  product  releases, 
each  of  which  undergoes  a  combination  of  formal 
verification  and  statistical  testing  techniques 
[Currit86,  Dyer87],  Regardless  of  the  approach  se¬ 
lected,  V&V  progress  must  be  tracked.  Require- 
ments/evaluatiod  matrices  {day  a  key  role  in  this 
tracking  by  providing  a  means  of  insuring  that  each 
requirement  of  the  product  is  addressed  [Powell86b, 
Sum86], 

5.  Problem  Tracking 

Other  critical  aspects  of  a  software  V&V  plan  are 
developing  a  mechanism  for  documenting  problems 
encountered  during  the  V&V  effort,  routing  prob¬ 
lems  identified  to  appropriate  individuals  for  correc¬ 
tion,  and  insuring  that  the  corrections  have  been  per¬ 
formed  satisfactorily.  Typical  information  to  be  col¬ 
lected  includes: 

•  when  the  problem  occurred 

•  where  die  problem  occurred 

•  state  of  the  system  before  occurrence 

•  evidence  of  the  problem 

•  actions  or  inputs  that  appear  to  have  led  to 
occurrence 

•  description  of  how  the  system  should 
work;  reference  to  relevant  requirements 

•  priority  for  solving  problem 

•  technical  contact  for  additional  informa¬ 
tion 

Problem  tracking  is  an  aspect  of  configuration  man¬ 
agement  that  is  addressed  in  detail  in  the  curriculum 
module  Software  Configuration  Management  [To- 
mayko87].  A  practical  application  of  problem  track¬ 
ing  for  operating  system  testing  is  presented  in 
[Sum86]. 

6.  Tracking  Test  Activities 

The  software  V&V  plans  must  provide  a  mechanism 
for  tracking  the  testing  effort.  Data  must  be  col¬ 
lected  that  enable  project  management  to  assess  both 
the  quality  and  the  cost  of  testing  activities.  Typical 
data  to  collect  include: 

•  number  of  tests  executed 

•  number  of  tests  remaining 

•  time  used 

•  resources  used 
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•  number  of  problems  found  and  the  time 
spent  finding  diem 

These  data  can  then  be  used  to  track  actual  test 
progress  against  scheduled  progress.  The  tracking 
information  is  also  important  for  future  test  schedul¬ 
ing. 

7.  Assessment 

It  is  important  that  the  software  V&V  plan  provide 
for  the  ability  to  collect  data  that  can  be  used  to 
assess  both  the  product  and  the  techniques  used  to 
develop  it  Often  this  involves  careful  collection  of 
error  and  failure  data,  as  well  as  analysis  and  classi¬ 
fication  of  these  data.  More  information  on  assess¬ 
ment  approaches  and  the  data  needed  to  perform 
them  is  contained  in  [Brown87]. 


Glossary 


acceptance  testing 

The  testing  done  to  enable  a  customer  to  deter¬ 
mine  whether  or  not  to  accept  a  system 
[IEEE83a]. 

correctness 

The  extent  to  which  software  is  free  from  faults. 
The  extent  to  which  software  meets  user  expec¬ 
tations  [IEEE83a]. 

coverage 

Used  in  conjunction  with  a  software  feature  or 
characteristic,  the  degree  to  which  that  feature  or 
characteristic  is  tested  or  analyzed.  Examples 
include  input  domain  coverage,  statement 
coverage,  branch  coverage,  and  path  coverage 
[Morell88]. 

error 

Human  action  that  results  in  software  containing 
a  fault  [IEEE83a]. 

failure 

Incorrect  behavior  of  a  program  induced  by  a 
fault  [Howden87]. 

fault 

An  accidental  condition  that  causes  a  product  to 
fail.  [IEEE83a]. 

integration  testing 

An  orderly  progression  of  testing  in  which  soft¬ 
ware  elements,  hardware  elements,  or  both  are 


combined  and  tested  until  the  entire  system  has 
been  integrated  [ IEEE83a >]. 

proof  of  correctness 

A  formal  technique  to  prove  mathematically  that 
a  program  satisfies  its  specifications  [IEEE83a], 

regression  testing 

Selective  retesting  to  detect  faults  introduced 
during  modification  of  a  system  or  system  com¬ 
ponent  Retesting  to  verify  that  modifications 
have  not  caused  unintended  adverse  effects  and 
that  the  modified  system  or  system  component 
still  meets  its  specified  requirements  [IEEE83a]. 

software  technical  review  process 

A  critical  evaluation  of  an  object  Walk¬ 
throughs,  inspections  and  audits  can  be  viewed 
as  forms  of  technical  reviews  [Collofello88]. 

system  testing 

The  process  of  testing  an  integrated  hardware 
and  software  system  to  verify  that  the  system 
meets  its  specified  requirements  [IEEE83a], 

testing 

The  process  of  exercising  a  system  or  system 
component  by  manual  or  automated  means  to 
verify  that  it  satisfies  specified  requirements  or 
to  identify  differences  between  expected  and  ac¬ 
tual  results  [IEEE83a]. 

unit 

Code  that  is  meaningful  to  treat  as  a  whole.  It 
may  be  as  small  as  a  single  statement  or  as  large 
as  a  set  of  coupled  subroutines  [Morell88], 

unit  testing 

The  testing  of  a  software  unit. 

validation 

The  process  of  evaluating  software  at  the  end  of 
its  development  process  to  ensure  compliance 
with  its  requirements  [IEEE83a].  Other  products 
besides  code  can  be  validated,  such  as  require¬ 
ments  and  designs,  through  the  use  of  prototypes 
or  simulation  [Howden87], 

verification 

The  process  of  determining  whether  or  not  the 
products  of  a  given  phase  of  a  software  devel¬ 
opment  process  fulfill  the  requirements  estab¬ 
lished  during  the  previous  phase  [IEEE83a]:  Of- 
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ten  equated  with  proof  of  correctness,  which 
proves  equivalency  of  programs  to  formal  speci¬ 
fications  [Howden87]. 
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Teaching  Considerations 

Suggested  Schedules _  Exercises  and  Worked  Examples 


The  nature  of  this  module  lends  itself  to  several  pos¬ 
sible  uses,  depending  upon  the  topics  of  interest  and 
the  depth  of  coverage  desired. 

Semester  Course.  This  module  can  provide  a 
framework  for  developing  a  graduate  one-semester 
course  on  software  verification  and  validation.  The 
outline  of  the  module  can  be  used  to  structure  the 
syllabus  for  the  course.  The  amount  of  time  to  be 
spent  on  each  topic  will,  of  course,  depend  upon  the 
background  and  interests  of  the  students  and  instruc¬ 
tor.  It  is  recommended,  however,  that  each  V&V 
approach  in  the  outline  be  addressed  in  the  course,  to 
insure  that  the  students  have  sufficient  breadth  to  un¬ 
derstand  the  context  for  applying  each  approach  and 
for  combining  approaches  where  appropriate. 

Overview  Lectures.  This  module  can  also  be  used 
as  a  basis  for  an  overview  of  the  software  V&V  area. 
Such  an  overview  could  be  presented  in  2  to  3  one- 
hour  lectures.  The  overview  could  serve  as  the  first 
week’s  lecture  material  for  a  course  developed  from 
one  of  the  more  advanced  curriculum  modules  in 
this  area,  such  as  Unit  Testing  and  Analysis.  This 
overview  would  also  be  valuable  for  management  in 
an  industrial  environment 


Any  course  on  software  V&V  requires  worked  ex¬ 
amples  and  exercises  illustrating  the  techniques 
being  taught  Some  suggestions  can  be  found  in  The 
Software  Technical  Review  Process  [Collofel!o88], 
Unit  Testing  and  Analysis  [Morell88],  and  Formal 
Verification  of  Programs  [Berzt»ss88]. 

A  useful  exercise  for  students  taking  an  introductory 
V&V  course  based  on  this  module  is  the  develop¬ 
ment  of  an  integrated  V&V  plan  for  a  project  The 
instructor  can  assign  teams  of  students  to  projects. 
Projects  can  address  different  application  areas,  pos¬ 
sibly  using  different  development  approaches.  For 
example,  one  team  might  address  V&V  issues  for  an 
expert  system;  another  team  might  be  concerned 
with  a  robotics  project  developed  with  incremental 
product  releases.  Presentations  at  the  end  of  the 
course  can  further  enhance  the  learning  experience 
for  all  involved. 
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